In U.S. Pat. No. 6,067,566, Moline teaches a method whereby a live musical performance, preferably encoded as well known Musical Instrument Digital Interface (MIDI) commands, can be sent over a network to many stations. The live performance can be selectively recorded or mixed with other pre-recorded tracks. The mechanism is a timestamp that is attached to each musical event (e.g. a MIDI Note-On command). By sequencing the timestamps from separate tracks, the tracks can be mixed. By delaying the mixing for at least the maximum expected delay of the communication channel, the (almost) live musical performance can be added to the pre-recorded tracks at a remote location. Further, a station receiving this performance can play along with the (almost) live performance. Moline is limited, however, in that the “play along” performance is not bi-directional. That is, a true jam session is not taking place. Moline suggests that a repetitive musical pattern could be established and enforced, and that jamming could take place by having each participant hear and play along with the others' performance from one or more prior cycles of the pattern. That play along performance is what would subsequently be heard by the others, during the next (or later) cycle. Such a constraint severely limits the range of artistic expression.
In U.S. Pat. No. 6,653,545, Redmann, et al. teach an alternative method and apparatus which permit real time, distributed performance by multiple musicians at remotely located performance stations. They show how the latency of the communication channel interconnecting the performance stations is measured and added to the behavior of a local electronic musical instrument so that a natural accommodation may be made by the local musician. Specifically, a local-only delay is introduced between the time that a musical note is played by the local musician at a performance station and the time that it is locally sounded. This delay is selected to be significantly representative of the delay inherent in the communication channel. However, the musical note is immediately sent to the remote performance station, and when received is essentially played immediately. In this manner, the notes are played at both stations at substantially the same time.
Timestamps
Moline above, and Neumann, et al. in U.S. Pat. No. 6,175,872 both teach the use of timestamps associated with MIDI data transmitted over a network as a mechanism for ordering the musical events and causing them to play at the appropriate time.
Moline requires that playback be held off for at least the maximum expected network delay in order to assure proper playback. This is not compatible with the requirements for a real time jam.
Neumann et al. identify timestamps as a means whereby musical events “from any remote site can be time positioned in the proper relative time sequence with respect to all the received MIDI data.” However, this does not enable a real time jam, except in special situations where “the network delays must be small enough to be insignificant to the playing.” Since Neumann et al. specify use of TCP/IP protocol, all musical event data will be received in order, however situations where a retransmission of a lost packet is required will seriously compromise a real-time jam. Neumann neither admits nor addresses this. However, Neumann does recommend the Network Time Protocol (NTP) as a means for synchronizing the clocks of remote stations contributing musical data.
However, even the well-regarded NTP is not entirely sufficient for synchronization. NTP is described in the specification RFC 1305—Network Time Protocol (Version 3) Specification, Implementation and Analysis by the Internet Activities Board of the Defense Advanced Research Projects Administration (DARPA). The RFC claims that NTP “provides the protocol mechanisms to synchronize time in principle to precisions in the order of nanoseconds.” Empirical testing suggests that NTP-based system clock synchronization as implemented in commercial operating systems such as Windows XP by Microsoft Corporation of Redmond, Wash. and Mac OS-X by Apple Computer of Cupertino, Calif. for personal computers exhibit both absolute time errors and significant drift. Their implementations of the NTP standards are wholly adequate for time-of-day functions, managing file directories and dating emails. However, combined with the hardware limitations of personal computers—especially those recently turned on or otherwise in a thermally unstable situation causing extreme clock drift—consumer grade operating systems commonly result in computer clocks which diverge from each other at rates of several seconds per day. This, in the real-time situation, represents drifts in excess of several milliseconds per minute. A drift rate such as this is incompatible with a need for time stamping real-time musical events for a remote jam, as within a few minutes one remote station may drift out of synch resulting in musical events arriving with timestamps apparently too old to be considered acceptable for live playback, even though this is not truly the case.
Bandwidth Limitations
Of musicians using the Musical Instrument Digital Interface (MIDI) preferred by both-Moline and Redmann et al., the majority employ a piano-style keyboard instrument. However, a variety of devices exist to allow the creation of MIDI events using or simulating other classes of musical instruments such as MIDI drums, electronic wind instruments (EWI) e.g. an electronic saxophone, electronic valve instruments (EVI) e.g. an electronic trumpet, and guitar-to-MIDI converters which adapt an electric guitar to generate MIDI events.
Though MIDI keyboards and MIDI drums usually generate a relatively moderate quantity of MIDI data, such is usually not the case with the other controller types. There is great expressiveness possible when combining fingering, breath, bite, and thumb controls on EWI and EVI instruments. Guitar-to-MIDI converters detect each of the strings separately, and follow the guitarist's bending of them individually. These non-keyboard and non-drum instruments commonly generate a larger number of MIDI events.
As the number of participants in a network jam increases, and as the average number of MIDI events produced by each participant increases, the aggregate traffic from a network jam may run into the bandwidth limits of one or more of the participants, resulting in more events being generated than can timely be received. A mechanism and method for controlling such an overload is needed.
Clean-Up
A side effect of such an overload will be that packets, if not substantially delayed, will be dropped. Further, the very protocols designed for low-latency real-time use, such as UDP/IP common on the Internet, are not reliable—typical figures would have one packet in one hundred being dropped. For whatever reason, a dropped packet can result in significantly undesirable performance: if a note-on event is missed, the note goes unheard; worse, if a note-off event is missed, the note is stuck on and sounds indefinitely.
There is a need to mitigate the effects of dropped packets both in real-time live performance, and in a performance captured for playback or manipulation at a later time.
Recording
Historically, recording studios are operated by an individual designated as the engineer. An engineer captures music made by musicians performing their art unfettered by the technical tasks associated with recording devices (the transport). The engineer supplies adequate blank media, advances or rewinds the transport to appropriate positions, selects certain channels for playback to accompany subsequent performances, and finally archives the “master” for duplication and later manipulation in the mixdown.
While such sophistication is not required to have a satisfying real-time jam experience, it is necessary if the remote performances are to be produced into a finished product.
A means is needed for providing recording studio-like functionality for a real-time remote collaboration.
MIDI Machine Control (MMC) is an established standard for manipulating the controls of a transport by using MIDI events. The standard is published in Complete MIDI 1.0 Detailed Specification by the MIDI Manufacturers Association, Inc. of Los Angeles, Calif. However, simply advancing MMC commands such as RECORD, STOP, etc. to remote stations, and making use of extant recording hardware or software is not adequate to provide a usable, collaborative recording environment. Available recording devices and software (also known as “sequencers” or “sequencing software”) are not aware of “lossy” channels such as expected in a real-time network jam. The cleanup mechanisms described below are not well served by prior art recording mechanisms. Further, the distributed nature of the remote collaboration calls for a similarly distributed transport mechanism to record locally the live performance of each musician, in full fidelity, and subsequently reintegrate those recordings into a master record of the musical collaboration.